feat: migrate SDK to v2 API by FrancescoSaverioZuppichini · Pull Request #13 · ScrapeGraphAI/scrapegraph-js

FrancescoSaverioZuppichini · 2026-04-14T12:34:27Z

Summary

Complete SDK rewrite for v2 API endpoints
Add ScrapeGraphAI({ apiKey? }) client factory (reads SGAI_API_KEY from env)
Add crawl namespace (start, get, stop, resume, delete)
Add monitor namespace (create, list, get, update, delete, pause, resume)
Add comprehensive Zod schemas matching API exactly
Remove generateSchema (no longer in API)
Rename: getCredits → credits, checkHealth → healthy

Usage

import { ScrapeGraphAI } from "scrapegraph-js";

// reads SGAI_API_KEY from env, or pass explicitly: ScrapeGraphAI({ apiKey: "..." })
const sgai = ScrapeGraphAI();

// scrape - minimal (defaults to markdown)
const { data } = await sgai.scrape({ url: "https://example.com" });

// scrape - multiple formats
const { data } = await sgai.scrape({ 
  url: "https://example.com", 
  formats: [
    { type: "markdown", mode: "reader" },
    { type: "screenshot", fullPage: true },
    { type: "json", prompt: "Extract product info" },
  ],
  fetchConfig: { mode: "js", stealth: true },
});

// extract
const { data } = await sgai.extract({ url: "https://example.com", prompt: "Extract prices" });

// search
const { data } = await sgai.search({ query: "best tools 2024", numResults: 5 });

// crawl
const { data: crawl } = await sgai.crawl.start({ url: "https://example.com", maxPages: 10 });
await sgai.crawl.get(crawl.id);
await sgai.crawl.stop(id);

// monitor
const { data: mon } = await sgai.monitor.create({ url: "...", interval: "0 * * * *", formats: [...] });
await sgai.monitor.pause(mon.cronId);

// utilities
await sgai.credits();
await sgai.healthy();
await sgai.history.list({ service: "scrape", limit: 10 });

Breaking Changes

All function signatures changed to match v2 API
Old v1 functions removed (smartScraper, markdownify, searchScraper, sitemap, etc.)
generateSchema removed
getCredits → credits, checkHealth → healthy

Test plan

bun run format - no fixes needed
bun run lint - passes
bunx tsc --noEmit - passes
bun run build - passes
bun test - 52 unit tests pass
bun run test:integration - 10 integration tests pass (includes format variations)

🤖 Generated with Claude Code

- Add all API request/response schemas matching v2 API exactly - Remove llmConfig from schemas (not exposed in SDK) - Add comprehensive types for all endpoints Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add scrape, extract, search, generateSchema endpoints - Add crawl namespace: start, get, stop, resume, delete - Add monitor namespace: create, list, get, update, delete, pause, resume - Add getCredits, checkHealth, getHistory, getHistoryEntry - Export schemas for client-side validation - Add zod dependency Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Replace cancelled with deleted in ApiCrawlStatus - Add deleted to ApiHistoryStatus - Move types from src/types/index.ts to src/types.ts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Update unit tests for new SDK structure - Add integration tests for live API - Fix schemas to use deleted instead of cancelled - Move types.ts out of folder Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…types - Add unit tests for all scrape formats (markdown, html, json, screenshot, summary, branding, links, images) - Add tests for fetchConfig options (mode, stealth, timeout, headers, cookies, country, scrolls) - Add tests for PDF/DOCX/image document scraping with OCR - Add extract tests for URL, HTML, and markdown inputs with schema - Add search tests with filters (location, timeRange, numResults) - Add crawl/monitor tests with full config options - Fix types to use z.input for request types (allows omitting fields with defaults) - Remove obsolete v1 integration_test.ts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Follow namespace pattern consistent with crawl.* and monitor.* Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Rename integration tests to *.spec.ts (excluded from CI) - `bun run test` runs only *.test.ts (unit tests for CI) - `bun run test:integration` runs *.spec.ts (live API tests) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Remove old v1 examples (smartscraper, markdownify, searchscraper, sitemap, agenticscraper) - Add scrape examples (basic, multi-format, pdf, fetchConfig) - Add extract examples (basic, with-schema) - Add search examples (basic, with-extraction) - Add monitor examples (basic, with-webhook) - Update crawl examples for namespace API - Update schema examples for camelCase fields - Update utilities for v2 response shapes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Update all API documentation for v2 endpoints - Add examples table with path and description - Add scrape_json_extraction example - Enhance scrape_pdf and scrape_multi_format examples - Update environment variables section Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Remove process.exit() from crawl example, use if/else instead - Fix non-null assertion in crawl example - Fix undefined variable references in README crawl section - Use consistent example.com URLs across all examples Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Require SGAI_API_KEY env var instead of fallback Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Rewrite all JavaScript code examples to match the new v2 SDK API from ScrapeGraphAI/scrapegraph-js#13. Key changes: - Replace factory pattern (scrapegraphai({ apiKey })) with direct imports - All functions use (apiKey, params) signature - scrape() uses formats array instead of single format string - Return type is ApiResult<T> with status check, not throw-on-error - crawl.status() renamed to crawl.get(), crawl.delete() added - monitor.create() uses formats array, not prompt - Restore generateSchema and checkHealth in docs - Schema params use JSON objects, not Zod instances - history is now history.list() and history.get() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Add ScrapeGraphAI({ apiKey? }) factory that reads SGAI_API_KEY from env - Rename client methods: getCredits → credits, checkHealth → healthy - Remove generateSchema (no longer in API) - Update all examples to use new client pattern - Update README with client usage Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Use new client pattern instead of standalone functions - Add test for scrape with no formats (defaults to markdown) - Rename tests for clarity Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Update CLAUDE.md to match Python SDK structure - Fix health endpoint path: /healthz -> /api/v2/health - Add playground script with .env loading - Track CLAUDE.md in git Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Change base URL to https://api.scrapegraphai.com/api/v2 - Rename SGAI_TIMEOUT_S to SGAI_TIMEOUT - Simplify health endpoint to use BASE_URL Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add ApiMonitorTickStatus, ApiMonitorTickEntry, ApiMonitorActivityResponse types - Add monitor.activity(id, params?) method for paginated tick history - Update monitor examples to poll activity and display diffs - Track seen ticks to avoid duplicate output - Use throw for proper TypeScript type narrowing Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

…Store Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Resolved conflicts: - package.json: kept v2.0.0 version, merged in husky/lint-staged from main - src/index.ts: dropped v1 deprecation warning (PR is v2 itself)

VinciGit00 · 2026-04-18T14:00:22Z

Test results (updated)

Conflict resolution

Merged origin/main into feat/v2-migration (commit ca8b357):

package.json — kept version: 2.0.0, brought in husky + lint-staged config from main, preserved the playground script
src/index.ts — dropped the v1 deprecation console.warn that main added (this PR is v2, so warning users to upgrade to v2 from v2 is nonsensical)
.husky/pre-commit — accepted from main

Local validation (all green)

bun install ✅
bun run format ✅ — no fixes
bun run lint ✅ — clean
bunx tsc --noEmit ✅
bun run build ✅ — dist/index.js 16.99 KB, dist/index.d.ts 34.18 KB
bun test tests/*.test.ts ✅ — 52/52 pass (290 expect() calls)

Live integration tests — ✅ all pass

Ran against https://sgai-api-v2.onrender.com/api/v2 (see follow-up comment for details). credits, healthy, scrape (default + fetchConfig), extract, search, history.list, and the full crawl.start → get → stop lifecycle all return status: "success".

Production api.scrapegraphai.com rejects the provided key as invalid — worth confirming deploy/promotion status before merging, but not a code issue.

VinciGit00 · 2026-04-18T14:02:29Z

Live integration results — all green ✅

Ran against https://sgai-api-v2.onrender.com/api/v2 (staging host — production api.scrapegraphai.com rejects this key as invalid).

Test	Status	Elapsed
`credits`	✅	146 ms
`healthy`	✅	44 ms
`scrape` (default markdown)	✅	—
`scrape` with `fetchConfig` (`mode: fast`, timeout 15s)	✅	273 ms
`extract` (prompt on example.com)	✅	1307 ms
`search` (numResults: 2)	✅	2694 ms
`history.list` (limit: 5)	✅	196 ms
`crawl.start` → `crawl.get` → `crawl.stop`	✅	—

Credits consumed during JS run: ~12 (487 → 475). The SDK handles success responses, nested namespaces (crawl.start/get/stop, history.list), and fetchConfig correctly.

Note: api.scrapegraphai.com/api/v2/credits returns 403 Invalid API key. with the provided key, so production is currently unreachable from this key — worth confirming prod deploy status before merging, but not a code issue.

FrancescoSaverioZuppichini and others added 17 commits April 14, 2026 14:12

feat(v2): add zod schemas and types for v2 API

55a7f42

- Add all API request/response schemas matching v2 API exactly - Remove llmConfig from schemas (not exposed in SDK) - Add comprehensive types for all endpoints Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

refactor: inline namespace functions directly in objects

5847b95

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix: use deleted status and move types to src/types.ts

a725c1c

- Replace cancelled with deleted in ApiCrawlStatus - Add deleted to ApiHistoryStatus - Move types from src/types/index.ts to src/types.ts Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat(v2): rewrite tests for v2 API

11b811c

- Update unit tests for new SDK structure - Add integration tests for live API - Fix schemas to use deleted instead of cancelled - Move types.ts out of folder Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

refactor: rename getHistory/getHistoryEntry to history.list/history.get

13578aa

Follow namespace pattern consistent with crawl.* and monitor.* Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore: split unit and integration tests for CI

6844d33

- Rename integration tests to *.spec.ts (excluded from CI) - `bun run test` runs only *.test.ts (unit tests for CI) - `bun run test:integration` runs *.spec.ts (live API tests) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: use local banner from media folder

22b936e

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix: remove hardcoded API key from integration tests

b5926d1

Require SGAI_API_KEY env var instead of fallback Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix: add node types to tsconfig

46bed0e

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore: remove .claude and CLAUDE.md, add to gitignore

276713b

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

minor changes

0738786

docs: reorganize examples table by service, add SGAI_API_KEY env var

3703f8c

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

FrancescoSaverioZuppichini and others added 10 commits April 14, 2026 15:23

test: update integration tests to use ScrapeGraphAI client

0522abc

- Use new client pattern instead of standalone functions - Add test for scrape with no formats (defaults to markdown) - Rename tests for clarity Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

chore: ignore scripts directory

7c7d31c

chore: bump version to 2.0.0

d4bad81

docs: update CONTRIBUTING.md for JS SDK

b9037f5

style: format package.json arrays

91f25ce

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

feat(examples): add polling loop to crawl examples

93a1b15

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix: update API base URL and rename timeout env var

2eba148

- Change base URL to https://api.scrapegraphai.com/api/v2 - Rename SGAI_TIMEOUT_S to SGAI_TIMEOUT - Simplify health endpoint to use BASE_URL Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

This was referenced Apr 15, 2026

feat: sync CLI with scrapegraph-js v2 PR #13 head ScrapeGraphAI/just-scrape#14

Closed

feat: sync CLI with scrapegraph-js v2 PR #13 head + consolidate commands ScrapeGraphAI/just-scrape#15

Closed

VinciGit00 mentioned this pull request Apr 15, 2026

feat!: migrate CLI to scrapegraph-js v2 API ScrapeGraphAI/just-scrape#16

Merged

6 tasks

VinciGit00 and others added 2 commits April 16, 2026 15:27

chore: link banner image to scrapegraphai.com and remove tracked .DS_…

50a59f9

…Store Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: merge main into feat/v2-migration

ca8b357

Resolved conflicts: - package.json: kept v2.0.0 version, merged in husky/lint-staged from main - src/index.ts: dropped v1 deprecation warning (PR is v2 itself)

VinciGit00 merged commit 7f9e5ba into main Apr 18, 2026
3 checks passed

VinciGit00 deleted the feat/v2-migration branch April 18, 2026 14:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: migrate SDK to v2 API#13

feat: migrate SDK to v2 API#13
VinciGit00 merged 29 commits intomainfrom
feat/v2-migration

FrancescoSaverioZuppichini commented Apr 14, 2026 •

edited

Loading

Uh oh!

VinciGit00 commented Apr 18, 2026 •

edited

Loading

Uh oh!

VinciGit00 commented Apr 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FrancescoSaverioZuppichini commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Usage

Breaking Changes

Test plan

Uh oh!

VinciGit00 commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test results (updated)

Conflict resolution

Local validation (all green)

Live integration tests — ✅ all pass

Uh oh!

VinciGit00 commented Apr 18, 2026

Live integration results — all green ✅

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FrancescoSaverioZuppichini commented Apr 14, 2026 •

edited

Loading

VinciGit00 commented Apr 18, 2026 •

edited

Loading